Ultralight: a Managed Network Infrastructure for Hep

We describe the networking details of NSF-funded UltraLight project and report on its status. The project’s goal is to meet the data-intensive computing challenges of the next generation of particle physics experiments with a comprehensive, network-focused agenda. The UltraLight network is a hybrid packetand circuit-switched network infrastructure employing both “ultrascale” protocols and the dynamic creation of optical paths for efficient fair sharing on long range networks in the 10 Gbps range. Instead of treating the network traditionally, as a static, unchanging and unmanaged set of inter-computer links, we instead are enabling it as a dynamic, configurable, and closely monitored resource, managed end-to-end, to construct a next-generation global system able to meet the data processing, distribution, access and analysis needs of the high-energy physics (HEP) community. THE ULTRALIGHT NETWORK A primary goal of the UltraLight Project [1] is to augment existing grid computing infrastructures, currently focused on CPU and storage, to include the network as an integral Grid component that offers reliable, and if possible guaranteed, levels of service. Developing and prototyping services to support this vision have been our focus, as we deployed and evolved the UltraLight network throughout 2005. The UltraLight network is shown in Figure 1. Figure 1 The UltraLight network UltraLight relies upon NLR, Abilene, ESnet, HOPI, UltraScienceNet (USNet), US-LHCNet and various regional networks (CENIC, FLR, MiLR) to create our UltraLight backbone network. UltraLight intends to provide on demand bi-directional data paths between UltraLight nodes. These paths will be either dedicated Layer 2 (L2) channels (with guaranteed bandwidth, delay, etc.) or L2 paths shared with other traffic. In both cases, the only constraint should be the Ethernet framing and the end-to-end connections will appear to be point-to-point. UltraLight will attempt to be as transparent as possible to end-users. In particular, users should be able to run the protocols of their choice over Ethernet. The use of dedicated L2 channels is an expensive solution, and often leads to poor utilization of network resources. L2 channels sharing the bandwidth available may be a more cost-effective solution and will have to be used where dedicated L2 channels cannot be provisioned. QoS mechanisms are being studied and deployed to improve the level of services (see the QoS/MPLS section following). The technology used will be based on tagged VLANs and/or MPLS but it should be transparent to end users. Some initial progress has been made (see MPLS/QoS Services and Planning) in conjunction with ESnet and the OSCARS and TeraPaths projects on developing QoS/MPLS capabilities which may eventually provide bandwidth management for the UltraLight infrastructure. UltraLight will dedicate a few L2 channels to connect each site and offers IPv4/IPv6 services. UltraLight has its own address space and autonomous system. We currently have the following network address spaces and services: • DNS domain ultralight.org; DNS at 192.84.86.88 • Autonomous System number 32361 • IPv4 addresses o 192.84.86.0/24 o 198.32.43.0/24 o 198.32.44.0/24 • IPv6 addresses 2001:468:0e9c::/48 • A network operations center (NOC) for problems is reachable via email at [email protected] * Dedicated L2 channels are provisioned by interconnecting waves or channels of time-division multiplexing (TDM) systems. A dedicated L2 channel is functionally equivalent to a circuit switched path. The network resource is reserved end-to-end and cannot be used by other traffic if under-utilized. The capacity of the channel cannot be temporarily extended; packets will be dropped if the traffic exceeds the channel bandwidth. Basic Network Services This allows us to interconnect the UltraLight testbed to conventional IP networks and facilitate access to the testbed from sites not connected to UltraLight. UltraLight peers with other backbones at Chicago, Los Angeles, Seattle and in New York. Rancid systems have been setup at Michigan and Caltech to track equipment configurations. An example is at http://linat08.grid.umich.edu/cgi-bin/cvsweb.cgi showing the type of configuration information tracked. The protocols used to control the information flow across the network are one of the important areas UltraLight plans to explore. The most widely used protocol, especially for reliable data transport, is TCP. TCP, its variants, limitations and extensions will be examined by UltraLight in conjunction with the FAST team [2]. TCP and its variants: TCP is the most common solution for reliable data transfer over IP networks. Since TCP was introduced in 1981, networks topology and capacity have evolved dramatically. Although TCP has proven its remarkable capabilities to adapt to vastly different networks, recent theories have shown that TCP becomes inefficient when the bandwidth and the latency increase. TCP’s additive increase policy (AIMD: Additive Increase, Multiplicative Decrease) for moderating the window size, based on the often-incorrect presumption that packet losses indicate network congestion, limits its ability to use the available bandwidth efficiently. The Ultralight testbed is the ideal place to evaluate and test new TCP stacks at 10 Gbps speed. Efficiency, the requirements and effect on end-hosts, the ability to coexist stably with other TCP implementations and the ability to share the bandwidth fairly will be evaluated. HSTCP, TCP Westwood+, HTCP, and FAST TCP are some of the new implementations we have tested. So far FAST has proven to the most promising, and adaptable to a variety of working environments. FAST TCP is an implementation of TCP with a new congestion control algorithm that is optimized for high speed long distance transfers. While the congestion control algorithm in the current TCP implementation uses packet loss as a measure of congestion, FAST TCP uses round-trip delay (time from sending a packet to receiving its acknowledgment). This allows FAST TCP to stabilize at a steady throughput without having to perpetually push the queue to overflow as loss-based schemes inevitably do. Moreover, delay has the right scaling with link capacity that enhances stability as networks scale up in capacity and size [3]. In November 2005 at SC2005, the UltraLight team sustained average data rates above the 100 Gbps level for several hours for the first time. The extraordinary data † A widely used network router and device monitoring system, see for example http://www.shrubbery.net/rancid/ transport rates were made possible in part through the use of the FAST TCP protocol, and a new FAST release. Other data transport protocols: Another approach to overcome TCP’s limitations is to use UDP-based data transport protocols. The best known protocol is UDT proposed by B. Grossman. Collaboration with the SABUL/UDT team is under discussion. Some servers dedicated to UDT tests have already been installed at CERN. Other servers may be installed at LosAngeles and directly attached to the UltraLight backbone. Network monitoring is essential for the UltraLight project. We need to understand our network infrastructure and its performance both historically and in real-time to enable the network as a managed robust component of our infrastructure. There are two ongoing efforts we are utilizing to help provide us with the monitoring information required: IEPM and MonALISA.